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Abstract 


The focus of this article is on the study of discrete optimal control prob- 
lems (DOCPs) governed by time-varying systems, including time-varying 
delays in control and state variables. DOCPs arise naturally in many multi- 
stage control and inventory problems where time enters discretely in a nat- 
ural fashion. Here, the Euler-Lagrange formulation (which are two-point 
boundary values with time-varying multi-delays) is employed as an effi- 
cient technique to solve DOCPs with time-varying multi-delays. The main 
feature of the procedure is converting the complex version of the discrete- 
time optimal control problem into a simple form of differential equations. 
Since the main problem is in discrete form, then the Euler-Lagrange equa- 
tion changes to an algebraic system with initial and final conditions. The 
graphic representation of numerical simulation results shows that the pro- 
posed method can effectively and reliably solve DOCPs with time-varying 
multi-delays. 
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1 Introduction 


It is well known that discrete calculus is an important tool for describing 
natural phenomenas, which is expanded from classic calculus [18, 3, 7, 13]. 
By employing discrete calculus in optimal control problem (OCP), also well- 
known as discrete optimal control problem (DOCP), one can uniquely dis- 
cover how to model natural phenomena. Discrete differential equations gov- 
ern the dynamics of a dynamical system in a DOCP are one of the newest 
exciting mathematical challenges [16, 21, 22, 12]. 


The primary difference between continuous and discrete-time systems 
arises from the necessity to convert analog signals to digital values, as well 
as the time required for a computer system to calculate and execute the 
corrective action to the output. 

A discrete time-control study on COVID-19 to address the quarantine 
and vital environmental loads has been explored in [2]. Mehracen et al. [14] 
proposed an approach to obtain the optimal solutions based on the Hamilton— 
Jacobi-Isaacs equation for the discrete-time nonlinear system by using neu- 
ral networks. In [11], the authors proposed an improved stability analysis 
method called a delay-mode-based functional method by weakening a con- 
dition in the Lyapunov—Krasovskii functional method. Adaptive dynamic 
programming as an effective intelligent control method has played an im- 
portant role in seeking solutions for optimal control. Approximate dynamic 
programming techniques are used to solve the value function, and hence the 
optimal control policy, in discrete-time nonlinear OCPs having continuous 
state and action spaces; see([1, 5]). The adaptive dynamic programming algo- 
rithm was introduced in [20] for solving infinite-horizon undiscounted OCPs 
in discrete-time systems. 


Discrete-time OCPs occur in many multi-stage control and scheduling 
problems, as may be expected. Originally, continuous-time OCPs can also be 
discretized suitably and subsequently formalized as discrete-time OCPs. Al- 
though due to the expansion of mathematical methods for solving continuous- 
time OCPs, this is not currently necessary. There are efficient methods for 
discrete-time OCPs in the literature. 

To solve combined discrete-time OCPs and optimal parameter selection 
problems concerned with general constraints, a computational method was in- 
troduced in [4]. The DOCP for discrete-time linear system control constraint 
was investigated in [23], in which the control input is a one-dimensional vari- 
able whose range is contained in a bounded closed interval. Li, Teo, and 
Duan [10] considered a class of DDTOCP that contains nonlinear inequal- 
ity constraints on both the state and control. In [19], authors discussed a 
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delay optimal tracking control for discrete-time systems with quadratic per- 
formance indexes when they are affected by persisting disturbances. 

This paper presents a novel approach to solving DOCP, including time- 
varying delays. A general formula of the structure for DOCP with time- 
varying delays can be considered as follows: 


ke=1 


JT(u(-)) = $7 F (xe, un, k), (1) 


k=ko 
subject to time-varying delay in a dynamic system 

Le = G( LK, Le—7,, Uk, Uk—w,s &), ko <k< ky, (2) 
with initial conditions: 


Lk = Ok, ko — Tho Sk < ko, 
Un = Ox, ko — Wry Sk < ko, (3) 


where x(-) is the state variable vector, u(-) is the control variable vector, k 
represents the time, F' and G are given functionals, kg and ky are fixed, ¢, 
and 0, are specific functions, 7 > 0 is delay function for state variable, and 
Wr = 0 is delay function for control variable. 

Whenever the associated dynamic system of DOCP depends on prior in- 
formation at a particular time, it can be considered that it is the DOCP with 
time-varying delays. A realistic distributed assumption, instead of a tradi- 
tional point-wise assumption, creates interesting cases of delays [17]. Discrete 
derivatives are essential for explaining physical phenomena with memories, 
as previous information about predators and even prey can have an impact 
on birth rates, rather than the current model of predator-prey relationships 
and hereditary traits; thus, DOCP with time-varying delays is applied to all 
physical processes with realistic distribution assumptions and experiences [6]. 
As it can be seen, the problem satisfying (1)—(3) includes the delay system. 
A delay system is a specific form of partial differential equation with infi- 
nite dimensions. Therefore, these types of mathematical problems are very 
important in engendering and physical sciences. 

Generally, time-delays systems can be found in control systems, lasers, 
traffic models, metal cutting, transmission lines, epidemiology, cell cycle, 
protein, production population dynamics, and neuroscience. Therefore, it is 
important to propose a beneficial method for solving time-delays systems. 
Also, solving optimal control problems is complicated in normal mode, espe- 
cially in non-linear modes. As a result, they become much more complicated 
in modes whose systems have time delays. So it is very valuable to work on 
such issues. 

As a review of this paper, the framework of this paper is organized as 
follows: 
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Section 2 includes the proposed technique for solving DOCP with time- 
varying delays in state and control variables. Finally, Section 3 contains a 
number of numerical examples that demonstrate the model’s effectiveness. 
We conclude in the last section. 


2 Main results 


There are several kinds of variational problems in calculus [9, 8]. Here, we 
propose the two-boundary value problem based on classical Euler-Lagrange 
equations to solve DOCP with time-varying delay. Therefore, we review 
some necessary definitions and theoretical concepts to derive our efficient 
technique. 


Definition 1. Suppose that x; (respectively, 7,41) takes on variations da, 
(respectively, 62,441) from their optimal values Z;, (respectively, Z;,41) satis- 
fying 


Ly = Tp t Org, Cee = Eepit OLK41. (4) 
Now with these variations, the performance index (1) becomes 


kp-1 
Fad Gish) = S) F Gnteniy’) 
k=ko 
ky-1 
J= J (Xr, Ko) = > F(&, + 6%, Beat + O@K41,k). (5) 
k=ko 


Definition 2. The first variation oJ is the first order approximation of the 
increment AJ = J — J. So, applying the Taylor series expansion of (5), we 
obtain 
“—! OF (Ex, Ena, k) OF (Zk, Fx+1,k) 
Uk, Uk+1; Uk, Ck+15 
6J = bx, 4 
oD Or, a OF R41 


OLR 41- (6) 


k=ko 


Theorem 1. For xz to be a contender for an optimum, the first variation 
of J must be zero on xx, that is, OJ(x,, dx) = 0 for all admissible values of 
dxp. This is a necessary condition. As a sufficient condition for minimum, 
we have the second variation 6? > 0, and for maximum, 5? < 0. 


Proof. The researchers can consider the proof in detail in (see [8, p. 37]). 


Lemma 1. Suppose that g, is a function in which the domain and range are 
each a discrete set of values. If g, is a discrete function satisfying 
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S- gnOX~ = 0, (7) 


where the function 6x, is discrete in the interval [ko,k,], then gx, = 0 for 
every k € [ko, ky]. 


Proof. Let gx, # 0 for some ko. Assume that x2, = 0 if s # ko and 6a, =1. 
Then 6 is a discrete function. In addition, 5°, drngx = gx, = 0, which is a 
contradiction. 


Definition 3 (Gateaux derivative). Suppose that X and Y are locally convex 
topological vector spaces, U C X is open, and f : X — Y. The Gateaux 
differential of f at u € U in the direction 7 € X, denoted by df(u;w), is 
defined as 


ag(usy) = jim LEER) Ory sew g 8) 


If the limit (8) exists for every y € X, then the function f is called Gateaux 
differentiable at u [15]. 


This paper investigates a structured strategy for finding the necessary 
optimality condition for the problem (1)—(3). It means that the DOCP with 
time-varying delays is analyzed in order to find the optimal control u(-) with 
the minimum performance index (1). Therefore, we investigate the necessary 
optimally condition of the DOCP with time-varying delays as follows. 


Theorem 2 (Necessary conditions for DOCP with time-varying delays). 
Suppose that the DOCP defined by (1)-(3) with ko, %,,, and ky is fixed. 
Also, suppose that X is a locally convex topological vector spaces, and that 
U Cc X is an open subset. In addition, assume that the following regularity 
conditions are satisfied: 


Rl. Lk, Lk, E Xx; 
R2. ux, Ur—w, € U3; 


R3. t : N — N and w, : N —> N are natural-valued functions, and 
T(-),w(-) > 0; 


R4. ko € Z, kp € Z, O: Z —>+ Z, and ¢: Z —> Z are known; 

R5. J is Gateaux differentiable at uz; 

R6. F and G are locally convex topological vector spaces. 
Then any solution u(-) € U must satisfy the following conditions: 


Nl. The state dynamics, for kg < k < kg: 


tea = Gast mstlnains k), ko Sk < ky. (9) 
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N2. The adjoint dynamics: 


OF 7 OG 
AP =0,k 
an An +2 44 a + R41 Vk 0,k > Tr, 
(10) 
OF OG 
Ab + Az =0 O.W. 
Oxy, Ee K+] OxrK , : 
where 
OG 
Wk — OLk—r, ’ 
F= F (xx, Ux, k), 
G= G(x, ee eae 
N3. The optimal control dynamics: 
OF OG 
Oun Ne Bg +X =0 Kk > wR, 
(11) 
OF 7 OG 
Buz * Guy — > ia 
OG 
where Nk = Oupny 
N4. The Boundary conditions: 
tk= Gk; k S ko, (12) 
Uk = Ox, k<ko, (13) 
OL GTi o i  te hk) te (14) 
OLk k=ky 


Proof. The required condition for the DOCP with time-varying delays is 
found by utilizing the variational method. Suppose that 


ky-1 
J(u(-)) = S- F (xx, Up, k) + are CC ae are k) san tes), 
k=ko 


(15) 
where (-) is the Lagrange multiplier. Let dx,, dup, OUp—7,, OUR—w,, and 


dA, be the variation of xy, Un, Le—7,; Uk—w,, aNd Ax, respectively. We then 
define a family of curves as follows: 
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Li = Le + Lp, 

€ a 
U4y = Teq1 + 6K 41, 
Lie, = Uk, + OLK-r,; 


16 
Us = Up + OUR, se) 
Uw, = Uk—w, + EOUR—wp 5 
Mest = Ag+ + €OAK41- 
Let 
L(k) — L (2k, Uk+1,Uk—rt,» Uk; We Sewk) 
== F' (ax, Ur, k) = ae (Gees teres Us they) my tes), 
(17) 
and 
L*(k) = L(x, Dees Pion Uk—w,> ®) 
= F(a;,, Up, k) + Nad) (G(x, Pipes UE Up w,>*) = wi41) ‘ 
(18) 
Also note that according to Definition 3, we get 
oe 6 = 
OJ (up; Oug) = lim Fue + edn) — Fun) 
e>0 € 
ky ks 
Lk) — L(k) d 
= lim ——2——— = —L*(k) |=. 1 
tim “OA Ly Fah) leo (19) 
k=ko ko 


The variational of functional J(u(-)) is given as 


kp-1 ky-1 
5 d OL*(k) dae OLS (k) dx, 4 
bJ(u()) = DL aL) leno= ae 
py de ; | Ox, de Oxi, de 
_, OL(k) dx, OL*(k) dug, 
Oxi, ,, de ' Out de 


OE) Mn, OEM) easy 
Ouy_., de Or, de ee 
(20) 


Also, according to (16), we have 


dx‘, 
de 


— OLk = OLk41, 


€ 


de 


U 
Jk = dug, 
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AN dt, a, dui. wy, _ 
ay = OXR+1; ae. = OLk—Tp 3 dé = OUR —wp + (21) 
Therefore, 
kp-1 
= - OL(k) OL(k) OL(k) 
dS (u(-)) = pS [ az Ox, + oa 6ER41 + One to 
OL(k) OL(k) OL(k) 
Ba One ee ee | a 


Also, we get from (17) that 


OL(k) = OF (xx, Ur, k) T OG (Xp, Uk—sy 5 Uk Uk—wy > K) 


Ox}, Oxy, - Ait OLk : 

OL(k) = yr OG (Xk, Ck—7y, 5 Uk Uk—wy > K) 

OL k—rp = er OLk— rp 

OL(k) - OF (ax, Uk; k) + \r OG(xr, Uk—t,» Uk, Uk—w, >) 

Our ~ Our Red Our 

OL(k) ar OG (&p, Cp—7,s Uk, Uk—w,s &) 

Oitiese, — AK+1 Our —wy, ’ 

ade = G(X, Tk—ry; Uk Uk—w,» &) — Taq, 

on) = —Ax41- (23) 
Uk+1 


Also, we can rearrange the term, including x;,4, in (22), as follows: 


Keot 


OL(by Thr» Tht1s Why Uh—-wi Ak+1) 5 
» Ox Th+1 
Se k4+1 
= OL (apts Theyre 1—ds Phys hy Whi 1-1 Ake) 5 
Ork; i 
DL (ko 1) Pho ~Te-1-1 Tho» Uhko~1 Uko—we—1-1 Ako) 
k 
OX ko ° 
kp-1 
OL re Gi he 9 Ty Ep aR gs oe) 
os oS 5) Chan 
x 
k=ko k 
OL(@e-1, Lk—tp1—-1) Chey Ue—1y Wh—wy_1—-11 Ak) hake 
=[ 6xrK] lh—ky 
OrK 
OL (Gia4 Wah io ilies Wee eh AR) 
Ds x 
k=ko k 
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We then conclude the first variation of J(u(-)) from equations (22)—(24) as 


k=k 
f 
brn] lh=ko 


= OL(fK_1,Lk—7,_ 1) Bk, Uk—1, Uk—w, 7-1) 
sJ(u(-)) =| (wp—1 k—Th1 —— k—wp 1-1 3) 


ky-l 
S (225, OL(@p—15©k— ry —1) Bhs UR—1s Uk—wp_ 1-1 Ak) 


+ 6 
k= KG chan : Oxy, ae 
aL(k) aL(k) AL(k) aL(k) 
Hy NEE Sey bux 4 bUR_—w, 4 5X : 25 
OXh—r, Sh k - Our ae OUk—wp, ve M OrR+1 +1) ( ) 


Therefore, the first variation is obtained as follows: 


= OL(@p-1, ©k—ry_y—1) Vhs UR—1; Uk—wp_ —1) Ak) k=k 
sJ(u()) =| er eo bee] load 
on oF 
k=ko 
+ ps1 (G(xe, Cer, Uk; Uk—w, K) — Be 41) 
0G 0G 0G OG 
aE ( bx, + Dux bux 4 mee btp—7), 4 Pie buku) (26) 
Let 
OG OG 
= = 27 
Ve OX kr, , He OUR—wy ( ) 
Since x, is specified function for k < ko, and 7 : N —N, then 
OLk, re, = 9, for all k; € [ko, kp — 1] and k; — 7, <0; (28) 
otherwise, 
N41, =0, hi — Th, > 0. (29) 
Similar to equations (28) and (29), we have 
6Uk;—we, = 9, for all ky € [ko, ky — 1] and k; — we, < 0; (30) 
otherwise, 
Neti: =9, kj — we, > 0. (31) 
Equation (26) can be rewritten as follows: 
7 OL =T) =F —1,“k; —1; Lipa = 
5F(u(-)) =| (ap 1,Uk—7,_1 -1) Uk, Uk-1; Uk—w,_ 1-1 k) barn] (ae 
Oxy, 9 
) 7 OG OF 7 OG 
DU (gag 8+ Mae) + (Gag + Mg) 
=ko 
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+ OAR41 (G(x, Lhe—tys Why Uh—w, K) — e+) 


+ Net 1VROTK—ny + Net MkOUk—we | : (32) 


In (32), the coefficients 6A,, 62%, and dug must be zero in order to gain 
the minimization of J(u(-)) and J(u(-)). Also, Euler-Lagrange equations are 
derived from (29) and (31) as follows: 


Ce = aise tin Up tes k), ko < Kk < ky, (33) 
OF 0G 
Apr + AZ T =0.k— 
a k Neri Dan + Anyi Vk 0, Tr > 0, 
(34) 
OF OG 
An + AE =0 O.W. 
Orr i +1 OXK ; | 
OF OG 
ae The NH Ba + Atle = 0, k- Wr, > 0, 
(35) 
OF \, oG 
Ou ae Ar+1 Bu, = 0, O.W., 
with the following conditions: 
Lk = Ok, ko — Tho Sk < ko, (36) 
Uk = Ok, ko — Wey Sk < ko, (37) 
OL (Spt Penis Ths UR AV ey 15 NR) 
— leckp= 0. (38) 


3 Numerical examples 


Some of the proposed features, including the efficiency and applicability of the 
technique, are discussed in this section with numerical examples. Our first 
example uses a non-autonomous DOCP with a time-varying state variable 
to implement the suggested method. In the second example, we also present 
the results of solving an autonomous DOCP with constant delays for state 
and control variables by the introduced method, indicating that we can solve 
optimal control problems with delays efficiently by this method. 


Example 1. Consider the following cost functional: 
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14 


T=) (xe + ug), (39) 


k=0 
subject to non-autonomous recursive equation with time-varying delays 
Cre = Apt, + Ai, Ler, + Brur, 0<k<14, (40) 
and the following condition 
rt, =1, k<0, (41) 


where 7, is the delay function satisfying 7 > 0 for 0 <k < 14, and A, =k, 
Aj, = 1, and B, = 1. The approach presented in this article has been applied 
to solve the DOCP with time-varying delays (39)—(41). The numerical results 
of this example are shown when Tt, = 3—k?. The Lagrange function L is 
defined as follows: 


L(k) =E (2p, 2e41,2e—r,3 Uns k) 


=r + uz + Ag+i (Lk + kavp—34k2 + UK — Lk+1)- (42) 


Therefore, the necessary conditions for the problem (39)—(41) are obtained 
as follows: 


Leo = LE + ke 7, + Ur, 0<k< 14, 
Qrp — An + Ansik + Anyi =0,k-3+k? <0, (43) 
2K _ Ak “le Ar+ik = 0, O.W., 


2up +Any1 =0, OS k< 14. 


Additionally, the following conditions contribute to obtain the solution: 


opi, SO, (44) 
Ais = 0. (45) 


The numerical results of state and control variables of Example 1 are 
shown in Figure 1 when t, = 3 — k?. Also, we show the convergence 
curve of the performance index function to illustrate the performance of the 
proposed method, in Figure 2. 


Example 2. Consider the following linear multi-delays time invariant prob- 
lem to minimize the following functional: 


1 100 1 
J(u) = 5 do (@e + suk): (46) 
k=0 


subject to 
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5 10 4s 
Time 


(a) State variable x, 


= ; — : 
2 gH 6 3 io 2 4 
ae / Time 
/ 


(b) Control variable ux, 


Figure 1: Approximation of state and control variable of Example 1. 


PI? 


Figure 2: The convergence of performance index function of Example 1. 
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1 
Ck = — Le + Lk—r + Uk — FUk—w, 0<k < 100, (47) 


and the following condition 


Note that in this example, 


The Lagrange function is defined as follows: 


LE) SL Ge tepig tina ky les ®) 


ug + Angi(—2k + Fk—6 + UE 5 Uk-8 Le41)- (50) 
The following equations give the optimal solution: 


1 
Lk+1 = —Le+ Le-6 + Up — =Uk-g 0<k< 100, 
Lp Ak — Angi + ARs4i =0,k-6<0, 


Lk Ak = Ak+1 = 0, O.W., (51) 
atk fe Ak+1 . 5 Ak+1 = 0, k-8 < 0, 
9 Uk = Ak+1 = 0, O.W., 


with the boundary conditions: 


te=1, k<0, (52) 
W=0.. 420, (53) 
Aioo = 0. (54) 


The analytic solution to this problem is not available. In Figure 3, the state 
and control variables of problem (46)—(48) are depicted. To demonstrate 
the performance of the proposed method, we show the convergence curve of 
the performance index function in Figure 4. 


Example 3. Consider the following two-Dimensional nonlinear time-delays 
autonomous problem to minimize the following functional: 


ky-l 
T(ur(-), u2(-)) = So (wt (h) + 23(k) + ui (k) + 03 (k)), (55) 
k=0 
subject to 
gi(k+1) = 23(k — 2) — 0.2u1(k), O<k<ky-1, (56) 
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Time 


(a) State variable x; 


uh) 53] f 


(b) Control variable wu, 


Figure 3: Approximation of state and control variable of Example 2 


PI 


07 ¢ 
wall 


y 
ost 


Figure 4: Convergence of performance index function of Example 2. 
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xo(k +1) = 23 (k — 2) — 0.2ue(k), OR :-b Shp 1, 
and the following conditions 


ay(k)=1, —-2<k<0, 
a(k)=-1, -2<k<0. 


The following equations give the optimal solution: 


L(k) =L(&x,0k41,2k—r, Uk; k) 


=a} (k) + 03(k) + ut (k) + ub(k) + Ar (k + 1)(23(k — 2) — 0.2u1(k)) 


+ Ao(k + 1)(x?(k — 2) — 0.2u2(k)) — Ar(k + 1)2i(k +1) 
— Ag(k + 1)x9(k + 1), 


From equation (34)—(38), we get 


OF ss aG 
t = -—2< 
Fath 7 M+ APR + Dg = 0, k= 250, 
OF aG 
Roe P.G RS 0: 
Baste) Ae ay 


( 201(k) — Aa(k) + An(k + 1)(2a2(k — 2)) =0,k -2 <0, 


221(k) — \1(k) = 0, 0<k-2, 


{ 2x0(k) — do(k) + Ao(k + 1)(2e1(k — 2)) =0,k -2 <0, 


2x2(k) — Ae k) =0, 0<k-2, 


oo 


2u2(k) — 0.2\2(k +1) =0,0<k—2, 


xi(k +1) = 23(k — 2) — 0.2ui(k), 


ao(k +1) = 22(k — 2) —0.2u1(k), 


with the boundary conditions: 


a(k)=1, —-2<k<0, 
a(k)=-1, -2<k<0. 
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(62) 


(63) 


(64) 
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(4) 


0 2 4 6 8 10 
Time 


(a) State variable xj (k) 


Time 


(b) State variable x2(k) 


Figure 5: Approximation of state variable of Example 3 


The analytic solution to this problem is not available. In Figure 5, the state 
variable of the problem (55)—(58) is depicted. Similar to the previous ex- 
amples, we show the convergence curve of the performance index function in 
Figure 7. Also, the control variable is illustrated in Figure 6. 


4 Conclusion 


By introducing a new Lagrange multiplier, the original DOCP with time- 
varying delays problem has been transformed into DOCP problems without 
time-delay terms to avoid solving the DOCP problem with time-delay terms. 
In this regard, we utilized the discrete method to derive the new Euler— 
Lagrange delay formula with a two-point boundary to solve DOCP with 
time-varying delays. It is important to give a way to solve DOCP with time- 
varying delays, according to its application. In this technique, we utilized 
the variation method to construct the Euler-Lagrange formula with a two- 
point boundary in order to solve DOCP with time-varying delays, which has 
not been done before. Moreover, two illustrations were supplied to demon- 
strate how the technique could be used. The performance index influenced 
the DOCP problem of discrete time-delay systems, and also an approximate 
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Figure 6: Approximation of control variable of Example 3 
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Figure 7: Convergence of performance index function of Example 3. 
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regulator was proposed. The simulation results showed that it is simple to 
implement and robust. 
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